Defect classification is an important part of steel surface defect detection. When the Convolutional Neural Network (CNN) has achieved good results, the increasing number of network parameters consumes a lot of computing cost, which brings great challenges to the deployment of defect classification tasks on personal computers or low computing power devices. Focusing on the above problem, a novel lightweight network model named Mix-Fusion was proposed. Firstly, two operations of group convolution and channel-shuffle were used to reduce the computational cost while maintaining the accuracy. Secondly, a narrow feature mapping was used to fuse and encode the information between the groups, and the generated features were combined with the original network, so as to effectively solve the problem that "sparse connection" convolution hindered the information exchange between the groups. Finally, a new type of Mixed depthwise Convolution (MixConv) was used to replace the traditional DepthWise Convolution (DWConv) to further improve the performance of the model. Experimental results on NEU-CLS dataset show that, the number of floating-point operations and classification accuracy of Mix-Fusion network in defect classification task is 43.4 Million FLoating-point Operations Per second (MFLOPs) and 98.61% respectively. Compared to the networks of ShuffleNetV2 and MobileNetV2, the proposed Mix-Fusion network reduces the model parameters and compresses the model size effectively, as well as obtains the better classification accuracy.